Method and apparatus for synchronizing audio and video in encrypted videoconferences

ABSTRACT

The invention provides a system that preserves the synchronization of the audio and video presented at a secure conferencing site without necessitating decryption, decompression, compression, and encryption of signals at the hub. The presently preferred embodiment of the invention provides an apparatus and method for synchronizing audio and video in encrypted videoconferences that comprises a plurality of conference sites; and a hub for receiving a compressed and encrypted, composite audio and video signal from each site, for determining a currently active site, and for transmitting said composite audio and video signal from said currently active site to all other sites; said hub receiving a compressed and encrypted audio only signal from each site; wherein said hub routes all incoming compressed and encrypted audio only signals to each site.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a divisional application of U.S. patent applicationSer. No. 10/661,110, filed Sep. 12, 2003, now U.S. Pat. No. 7,464,262,the entirety of which is incorporated herein by this reference thereto.

BACKGROUND OF THE INVENTION

1. Technical Field

The invention relates to videoconferencing systems. More particularly,the invention relates to a method and apparatus for synchronizing audioand video in encrypted videoconferences.

2. Description of the Prior Art

In many video conferencing systems, it is possible to conduct aconference involving more than two conference sites. In suchconferences, the network topology often incorporates a hub that receivesincoming audio and video signals from each of the participating sites,and routes appropriate outgoing audio and video signals to each site.Because each site typically has a single display on which to presentvideo signals routed from the hub, a single video signal is routed fromthe hub to each site to conserve bandwidth. However, unlike video, audiofor more than one site may be presented simultaneously at a given site,and indeed conference participants at a given site viewing a singlevideo signal may still benefit from hearing audio originating from allconference sites.

Existing systems meet this need by mixing audio signals and selectingvideo signals at the hub. All audio signals received at the hub aremixed together and routed to each site. However, only the video signalthat a particular site is to display is routed to that particular site.The audio mixing and video selection operations are sufficiently simplethat the latencies introduced into the audio and video signals arecomparable. The audio and video presented at the destination site aretherefore synchronized.

In the case of a video conferencing system incorporating encryption,several challenges are encountered. If the standard approach is to beused, the video and audio signals must be decrypted and decompressedprior to audio mixing and video selection. This leads to a substantialincrease in latency. Further, it requires that the physical site housingthe hub be secured and authorized to handle unencrypted information.

An alternative approach involves sending the audio signal received fromeach site to each other site. However, in this approach each site mustthen decrypt and decompress the audio and video signals separately. Mostnotably, the audio signal originating from the same site as thedisplayed video is handled separately from the displayed video. Thediscrepancy in latencies that results produces a desynchronization ofthe audio associated with the displayed video. The result is aconfusing, distracting, and unsatisfying experience for the conferenceparticipants.

It would be advantageous to provide a system that preserves thesynchronization of the audio and video presented at a secureconferencing site without necessitating decryption, decompression,compression, and encryption of signals at the hub.

SUMMARY OF THE INVENTION

The invention provides a system that preserves the synchronization ofthe audio and video presented at a secure conferencing site withoutnecessitating decryption, decompression, compression, and encryption ofsignals at the hub. The presently preferred embodiment of the inventionprovides an apparatus and method for synchronizing audio and video inencrypted videoconferences that comprises a plurality of conferencesites; and a hub for receiving a compressed and encrypted compositeaudio and video signal from each site, determining for each conferencesite a currently displayed composite audio and video signal, andtransmitting each currently displayed composite audio and video signalto each respective site; said hub receiving a compressed and encryptedaudio only signal from each site; wherein said hub routes all incomingcompressed and encrypted audio only signals to each site. The inventionfurther comprises an audio deselection and mixing device located at eachconference site that deselects the audio only signal corresponding tothe currently displayed composite audio and video signal and mixes allother audio only signals with the audio signal within the currentlydisplayed composite audio and video signal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block schematic diagram showing a system that implements amethod and apparatus for synchronizing audio and video in encryptedvideoconferences according to the invention; and

FIG. 2 is a block schematic diagram showing a video conference locationthat operates in connection with a method and apparatus forsynchronizing audio and video in encrypted videoconferences according tothe invention.

DETAILED DESCRIPTION OF THE INVENTION

The invention provides a system that preserves the synchronization ofthe audio and video presented at a secure conferencing site withoutnecessitating decryption, decompression, compression, and encryption ofsignals at the hub.

FIG. 1 is a block schematic diagram showing a system that implements amethod and apparatus for synchronizing audio and video in encryptedvideoconferences according to the invention. In the preferred embodimentof the herein disclosed conferencing system, each of sites A-E, 11, 13,15, 17, and 19, respectively, sends to the hub 10 a compressed andencrypted, composite audio and video signal. For each of the sites, thehub determines a currently displayed composite audio and video signal,based upon conference control information, and sends this compositeaudio and video signal to each respective site without decompressing ordecrypting the signal. There is no global active site. Instead, it isunique to each site. Thus, each site gets its own currently displayedcomposite signal.

Each site also sends to the hub a compressed and encrypted audio onlysignal. It should be noted that the audio only signal sent from eachsite may in fact be a mixed audio signal composed of audio obtained fromseveral microphones at a single conferencing site. The hub routes all ofthe incoming compressed and encrypted audio only signals to each site.

FIG. 2 is a block schematic diagram showing a video conference locationthat operates in connection with a method and apparatus forsynchronizing audio and video in encrypted videoconferences according tothe invention. Each site, such as the five seat audio-videoteleconference center 11 shown in FIG. 2 a, decrypts, decompresses, andthen displays the video within the composite audio and video signalreceived from the hub. The actual technique used forencryption/decryption and compression/decompression is a matter ofchoice to the person skilled in the art and is, therefore, not discussedin detail herein.

The signals transmitted to and from each site typically compriseconference control signals 22 to coordinate feeds and switching via anout-of-band mechanism such as an intranet or the Internet; a locallyselected compressed and encrypted composite audio and video output 23; acompressed and encrypted audio only output preferably obtained by mixingseveral microphone feeds obtained at the site 24; a compressed andencrypted primary view composite audio and video input 25 selected bythe hub control; a compressed and encrypted secondary view compositeaudio and video input 26 selected by the hub control for split screengeneration (see the discussion below); and n lines of compressed andencrypted audio only inputs 27 which correspond to each site in theconference.

The audio from the composite audio and video input signal, together withthe other, separately decrypted and decompressed audio only inputsignals, is passed to an audio deselection and mixing device 21 (FIG. 2b). The separate audio only signal corresponding to the audio signalwithin the composite audio and video input signal is deselected by thedevice using a logic control signal 28 generated by an executivecontroller 12 (see FIG. 1). The logic control signal is shown in FIG. 1as an out-of-band signal C2 generated by the executive controller, i.e.the hub controller, based upon video selection signals within thesystem. See Table 1 below, which details this exemplary audio selectionlogic scheme. Note that Table 1 shows the audio from the composite audioand video signal for the sending room in an upper cell of each receivingroom row and the combined audio only signals from which the sending roomaudio has been subtracted in a lower cell of each receiving room row.For example, the rows for receiving room A intersects a column forsending room B in which the audio from the composite audio and videosignals for sending room B is shown in an upper cell and the combinedaudio only signal from which the audio for room B has been subtracted,i.e. rooms CDE, shown in a lower cell. Those skilled in the art willappreciate that any known technique may be used for the audiodeselection process.

The other audio signals, including the audio from within the compositesignal, are mixed together and reproduced at the conferencing site. Thisprocess ensures that each audio signal is reproduced only once. Becausethe audio and video within the composite audio and video signal aretransmitted, decrypted, and decompressed together, the latenciesintroduced into the signals are well matched.

TABLE 1 Audio Selection Logic SENDING ROOM A B C D E RECEIVING A — B C DE ROOM — CDE BDE BCE BCD B A — C D F CDE — ADE ACE ACD C A B — D E BDEADE — ABE ABD D A B C — E BCE ACE ABE — ABC E A B C D — BCD ACD ABD ABC—

The audio associated with the displayed video is therefore synchronizedwith the displayed video. Because the audio signals transmittedseparately are processed separately, a latency different from that ofthe composite signal may be introduced. However, because these audiosignals are not associated with the video displayed, this discrepancy isnot noticeable to the participants. Nonetheless, the audio deselectiondevice may be equipped with delay circuitry to attempt to better alignthe separate audio signals with the composite signal.

If a split screen display is to be presented at a site, the hubtransmits two composite audio and video signals to the site. Followingdecryption and decompression of the composite signals, the site uses asplit screen composition processor to compose the split screen displayfrom two video signals. In this case, two audio signals are deselectedusing the audio deselection device 21.

The audio deselection hub may also be used to deselect those audiosignals not directly associated with the ongoing conversation. This mayhelp in reducing the sense of background noise and audio clutter oftenobserved during conferences where several audio signals are mixed.

Although the invention is described herein with reference to thepreferred embodiment, one skilled in the art will readily appreciatethat other applications may be substituted for those set forth hereinwithout departing from the spirit and scope of the invention.

Notably, while the invention is describe with respect to a secureconferencing system incorporating both compression and encryption, theinvention is also useful in conferencing systems incorporating onlyencryption, only compression, and neither encryption nor compression. Insystems incorporating only encryption, the invention obviates the needfor securing the conference hub. In systems incorporating onlycompression, the invention reduces the total system latency. In systemsincorporating neither encryption nor compression, the invention ensuresoptimal synchronization of audio and video signals.

Accordingly, the invention should only be limited by the Claims includedbelow.

1. An apparatus for synchronizing audio and video in videoconferences,comprising: a plurality of conference sites; and a hub for receiving acomposite audio and video signal from each site, determining for eachsite a currently displayed composite audio and video signal, andtransmitting said currently displayed composite audio and video signalto each of said sites; said hub receiving an audio only signal, separatefrom said received composite audio and video signal, from each site; andwherein said hub routes all incoming received audio only signals to eachsite separately from said transmitted composite audio and video signal;each site comprising: an audio deselection and mixing device fordeselecting an audio only signal corresponding to an audio portion ofsaid currently displayed composite audio and video signal, and formixing said audio portion of said composite audio and video signal forsaid currently active site with all other audio only signals at saidsite; wherein audio associated with displayed video is synchronized withsaid displayed video.
 2. The apparatus of claim 1, wherein said audioonly signal for a site comprises: a mixed audio signal composed of audioobtained from several microphones at said site.
 3. The apparatus ofclaim 1, wherein said composite audio and video signals are encrypted.4. The apparatus of claim 1, wherein said composite audio and videosignals are compressed.
 5. The apparatus of claim 1, wherein saidcomposite audio and video signals are both encrypted and compressed. 6.The apparatus of claim 5, each site comprising: a decoder for decryptingand decompressing video within said currently displayed composite audioand video signal.
 7. The apparatus of claim 1, wherein said audio onlysignal from each site is encrypted and compressed, each site comprising:a decoder for decrypting and decompressing said compressed and encryptedaudio only signal from each site.
 8. The apparatus of claim 1, saidaudio deselection and mixing device further comprising: delay circuitryfor aligning said audio only signals with said composite audio and videosignal.
 9. The apparatus of claim 1, wherein said hub transmits at leasttwo composite audio and video signals to each site to provide a splitscreen display at each site.
 10. The apparatus of claim 9, wherein thoseof said audio only signals which correspond to said at least twocomposite audio and video signals are deselected at each said site. 11.The apparatus of claim 1, further comprising: an audio deselection hubfor deselecting those audio only signals not directly associated with anongoing conversation.